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. The/ 1^ Ilti\itaU6^^ Ckntf tteri^ on Testirigprdblrfis; sponiored 
by E4urtUww^ in YWc City at the j 

'r^Itee^dt Htobd Ai to i^V^* on OQtobar;30. Ih^iyM u 

Sri ■ amin^Qg ti^e i ^^^^r^oe ptop aon/ it wa'i. ^ nec^^ary , to toy to 
W#Pf^ fr^ years ^e at^ 

' tradUM hi^ i^iA^^ tq oonvwpOA paroportionpi Morej man 1000 : \ 

fiavifations^ BmtM% am s3tikc^±M Ql ^^ose favited are uiually r >^ ^ 
l^^tnt tie ft^ jj^rofiteDa, mM, ^vw tb ^fary to ja^^we somefliing of 
;,tibe Qi^^al ^^cur o at tiie saina time > 

\f ' tiy to arrwgTV^o^Bm whi^ W0idd; A^^ed tojsdme 50p-odd j^ple . 
of Avei^e tot^ests attil ba^k^nom * . * ^ 

' At l^e te€^i^ 1^^^ meisurrtaent AeoSy is imdergotog rapid re- : 
' v^€m unde^ £e imbaet oii ^msavmoMoui tneoi^s inf ora&ation Uieo^, ^ 
s^Wical yepiiiott ™^ 

§^e iheofyi to nami only a| few. These AeoiW are aU hi^y mathe-. 
mati^l, wid 4e braich^ pt mathtoatics^hich ftiey use areitio ^ 
jtooiPm tc^m^ m edu^tional and psychological measurement 

In tihe bioadCT flald of evaluations moreover, clinicaT piychplpgists ailfl - 
social psyoholo^stsj are deve^ping new m6tiiodi ' of as^^sing such ^ f 
things as creative tyents. p^onaHftlr traits^ tfie dim^sions ^ fff^oup A 
interaction, tiie natoe and duality of leadership in various settings, and 
the prooefses of humim judgment ITieie methods are in many eases 
, quite dffieifient from tiiose ^ployed in assessing dDgoitive aptitudes 
and sdiool ayiievement It was felt ttiat the ney/ meories asd methods 
\9 should be brought to the attention of measurement workers deapite.tiie 
Y i^ti*^ OTOsiderabie obstaol^ to rffe^ 
problem was to tty to reduce fteie obstacles as much as possibla, and ^ 
to find speakers who could present some of &e new tiieories and me^= . ^ 
ods in tenns' which measurement worKe^^ ^ 

W© were^ fortunate in securing as our luncheon sp^ker ftr, Daniel 
Starch,\whose addtessi . . And Iteve Not jVisdom, ' recalled forcibjy 
the n^d to teach students how to make di^icfl value Jud^eiits, and 
the need tp develop metihodi for measurin^tiie a^tt^^^ '9 
edumtional objective. / f _ . - ^ ' 

TTie rest of me program coniistea of a,flrst riiorning session on some . 
applications of . formation theory *to ^testing problems, two parallel . 
sessions l&terjba the morning, one on recent advances in psychomeWc* 
methi^s and one on^e evfiluation of group interaction, and an after-f 
noon session on new devrtopments in me education of abler students. 
^We hope that ftls program achieved the balances implied by its objec- 
tives^ ^ ' y ' ■ \' ' \ v. 

The Chairtaan welcomes this opportunity to express his sincel^ 
ap^eciation tb all tiie speakers for mejr contributions, to^ducatidnak^ 
Testing Service for sponsoring die Cajference, to*J^ck K. Rimal^ver 
for his unfailmg' support, assistance, and counsel,-afid to Mrs* Catii'' 
erine p, Sha^ for ner assistance in making such eAellent local ar- 
rangements. ^ . ' ^ * ( 
■ Edwahd E. CuhetOH, Cftdinnan * ^ 
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Multiple Assignments of Persons to Jobs^ 

frAUL S, DWYER 



1. InbDdiiction. t want to talk to you ^about a problem wl^gb ariiep 
when men are to beiassignad to jobs in the most efficient manner* In 
tiie tim^ at my dis{wsal I can give you only a brief outline of Ae nature 
of the problemj of the general methods proposed for its solution, and 
of my recent work on tiie solution of the problem with the use of toans- 
formations. However, I am giving you'^pplementary material which 
you may examine later in*more detail if yop ^desire to dbtain a more 
comprehensive View of the ^oblem^ and the me^ods of solution. 

2. The Natmre of ^e ^oblem>A simple iUuitratibn may serve tt give * 
you some idea df the nature of the problem. A corporation hires 4 
college graduates to fill 4 vacancies without determining which mdi- 
vidua! is to be pjaeed on which job. These graduates, diough tiiey have 
dififering abilities as indicated by thei? records, are alL hired for tiie 
same f salary. The probjem of the corporation is then to plapa these 4 
men on the 4 Jobs in such a way that the coi^ration^ will obtain maxi- 
mum value from their services. 

The coloration may do this by estimating the worth to the colora- 
tion in thousands of dollars per year of each individual if he were to be 
placed in (each one of die 4 jobs. Sudi a set of estimates is showa in 
Table L The entries in the table show the values, denoted by cij, which 
indicate die estimated contribution to the total eflFort (in units of 
$1000) which individual i will make if he is placed on job j. Thus, 
individual 1 is most valuable on job 1 but so is individual 2 and indi- 
vidual 4. The problem i^ to^lace all 4 individuals on all 4 jobs in such 
a way tha^the sum of the assigned Cij values is as large as possible. 

Now since e^ch individual can fill one and but one job and since 
each job must be'fi&d by soipe one individual, it follows that any 
assignment of the 4 individuals to the 4 Jobs must involve one an^^t 
one selectiog from each row tod from eadi column. Hence the problem 

*Thii rasearQh was iupported in part by the United Statss Air Force under Con- 
tract No. AF18(^) — 1050 monitored by Director, Detaclmient 4 (Crew Re- 
search La^ratory), Air Force Personnel and Training Research Center, Randolph 
Au- Force Base, Randolph Field. Texas. Pennission is panted for reproduction, 
tfanila^n, publicatfon, use mud oispoial in ^ole and in part by or for the United 
States Govemmenti ^ / ^ 
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bTOopes OTS ^ maldng selections of cij vdues, one from row 
and one from eaoh column, so ftat flie sum ia as iMge ai pos^le. 

A more general statement of the problem results from Ae liisplay 
.*of Table II wh^ Ae 0^ values of N jobs-andl N indiwduals are tadf- 
cated. !^© problem la to select N values of cij id^at tike sum of tiie 
eir^rolues-io-seleGted^to^as-lwp — 1^ 



3; Alternate Fomf of Ae Problemsr The form of T^ble I and Table II 
' is tiiat of a square aiTay of Cu values featuring N rows and N oolumM. \ 
As a rasult of pouping the number of columns^may be re4u^» ^ 
somte probltos, so that tiie ajxay taWs on r^tangular form wi& hei^t ; 
^ater^hm widA/For axmple tiie proble^. of Table I may be so. 
r^uced sinee die values ctf Cij m' column 3 are jdenticrt wi& Ae Cij 
values of wlunm 4, In^so f^ as the solution is cpni^med, there is no 
differenof ^between job 3 and job 4 so that die two jobs may be pouped 
tpgether if. a common job eategj y. Jf we denote, 4e number^f sudi^ 
cat^ories by m, and ti^e j^mwFof Jobs in Job category J by qj, wa 
have m — 3, ql - qs = 1* qs - 2 for ttie problem of Table L Ha , 
values qj, which indieate 4e numbers of individual to be assipied 
to ^e* respective job categories, are caUed quotas, IWs form of the 
prpblm which features these Job categories and quotas is sometimes 
known as the quota form. 

Tte quota form of tiie problem of Table I is shown in Table IIL 
TTi© quota fonn oftthe general problem of Table II is shpwn in 
Table TV. / - ~ ^ 

" Rows may abo be grouped to form personnel categories when the Cij 
values in different rows are identical or approximately so. /W^en ppr-^ 
sonnel categories and lob categories are both used, we have a two-way 
'grouped dlsWbution which takes on the forih sometmies^ed^e fr^ 
quency form. The number of personnel ^tegories is ta^n as n, and 
the frequency in personnel category i is indicated by fi^ cou^e the 
sum of Ae fi values equals &e sum of the qj values which to Nv This 
frequenoy fom of the problem is illush-ated In Tabto V, 

The form of Table I and Table II, Since it features tiie nonj^uped 
valu^ for botfi individua^;and jobs, is sometimes called the iAOT^idual 
fonn* " ^ '^^^^ ^ 

4. Equivalent Prolblems. Hiis problem is essentially tibe equivalent of 
problems in oAer flelds/ For example the Hitchcock trans^rtation 
problem is tiie maAematical equivalent of the personnel classiflcatiQn 
^blffln tough it mXh for the^selection of Ae Cj j values from each row 
and column so as to^mipimize, rather tiian maximize, the sum. There 

u 
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■■ • ^ . ^ ^ i , : 

is no her@ f a dlsotyll|i|QQ of ^ese ^uivaleEit problaoi but rtfajr^ 
B^m B us prch^dad for Aose who bib tet^reated. " 

V 5* MpAods of Solutio& Fri^ouily tTsed« Muiy meAoda ^^va ^^i^ 
pxop^pd for die iolutton oi Ail probjtoixyl Mono^^phs giving lurvey^ 

mi^odi' Ls^^M mU vyou^^^bentioq to &e me Aod pf aU possiblj 
aaaig^rati/the ilmpli^ niadio^ ^d*die metbod^^ o^ ojgtut^ regio^. 

fhe mtt^ of poiiible ^si^utteats, every p^sibla^lten^tive 
asii^p^t is ma^at Now Atre are^l &ltemative aisigoments diou^ 
some of tibe ^^cment iuma toiay be equal, Hua with N ^4 in 
Table % ^e ^mputotion of ^e 41 ^ 24 poisl^ bw^ diowu us the 
mftrifniim sum of $^,000* But ^e^mathod of all poisible assipiments is ^ 
impractical for hunger values of N sm^ Nl iiiCTeas^ ve^ rapidly. 

The simple ma Aod was dasigpied for more general problems In ' 
linei^ propamming and gBme ttieory, Danbig and Vbtaw ^ve ap^ 
pUed it to Ais probl^ (see D), It is jny ^mm Aat Ae madiinery of 
Ae shnplex method ii tu^ecessarily Domplek for problem ud Aat 
simpler methods, dosaib^ briows ar© prrferable. 

The ma&od of optimaljre^ons i4, very useful m solvmg the classiflea- 
tion problem espaciaUy whmi Ae number of job oategories is small 
(see E), But the meAod I vi^h to discuss with you tpday-is me^od 
of tirrasfomations bjn^fm die anray of Oij.valu^ cmi be transformed 
to a n4w wrriy from which solution ^n Jjf obtained by the seleotion 
of zero tems* 

6, Method of Transformations. A solution of ^e profal^, and there 
may be more t^Bn ohe,' ^nsiste m the assignment of each individual i 
to some job j so that the sum of die Cu valuarf^i to he mdixmiized. His 
« mftTiTniiiii simi is not die solution^ though it cBki be calculated race ^e 
solution U Imown/ A solutidn consist in ^e assignment, i.e.^ the pairs 
of values of i and Jlius in Table I a solution consiste m the assignment 
of man 1 tp job 1, naan 2 to job 3, man 3 to job 4^ and man 4 to job 2/ 
If we indicate the job assi^ed to individual i by Jt and consider the 
men in the order 1^ 2^ 3, 4 we can write tiiis solution compactly by 
Ji ^ 1, 3, 4 2. the solution sum is6 + e + 5 + 6^23 units but 4is 
is not the solution. The solution is ^mply the set of element (i, Ji), 

I ^n now sMe an impoittnt relation wUch tei^m as Ae basis of tihe 
mei^od of tmnsfomiations. Any cons^t may be ' subtracted fro^ 
every elwnent in' any row or ^lunm WiAout changing ihe solution, 
solution stm is deereased by ^e amount subtm^ed but the s6lu- 
tion is not changed. Hence we may subb'act iunultaneously constant 



^ Ciirdm every row i and constants Cj from every row j without changing 
"the sblution, The purpose of the metliod of transformations is to make 
use of successive subfraction& from roiv^ and columns until an array 
results from whicb di^ solution k imigiediately obtainable, Speciflc 
/ dirictiQns follow, * 

The first step in t^e solution of a maximization problem is the sub- 
.iraction of the largest element in ^ch roW from each element in that 
row. The insults of these subtraetions c^J<}^ are either 0 or negative, ^ 
The process is illustrated Mn Tabled where the maximum v^ues for 
? T ' the row^ of Table I are shown at the right of the first array and tfie 
values of Cij^^^ in the second array. Now the Cjj^'^ array, since it con' 
only hon-positive termsr cannpt have, a solution witfi a sum 
* ] greater Aan.zero. We cannot locate a solution with sum.0 as long as' 

mny column has all noh-^ero tewns. So we subtract the largest element^ 
' in each cohimn as indicated at the bottom of the array. The resulting 
Cij*^^ arrayihas at IWast on^e ^ero in «ach row and each columti," See the 
^ third array of Table VI. Sometimes a solution can be obtained from 
this array by using only 0 elements. This is not possible in Table VI, 
An addrtiorial ti^nsformation is in order, * 

Before indicating the nature of this transformation, we note that the 
second and third arrays of Table VI feature negative signs. TTiese 
could be eliminated if, in the first arrays the elements were f iibfracted 
from the maximum values rather than vice-versa, Then the subtraction 
of the. smallest eleiTieut in each column of the second array is indi- 
catedr This process is illustrated in the first three arrays of Table 
VII, The positive elements of the second and third of these arrays are ^ 
identical, aside from sign, with those of Table VI. 

We are unable to find a solution using the 0 elements of the third 
array of Table Vfl since the 0 terms in columns 2, 3, 4 are all in row 3. 
However if we' subtract — 1 from this row, we can then also subtract 
1 (the smallest non zero value in columns 2, 3 and 4) from each ele^ 
merit of columns 2, 3 and 4 to form the Cu*'^' array. The net sum of the 
subtracted values is 1 + 1 + 1 = 1 = 2 which is placed in the lower 
right corner, In general any such transformation in which the sum 
of the subtracted constants is positive and which does not result in 
negative terms, may constitute the next step of the solution. 

In this problem the solution can be obtained from the 0 terms of the 
Cij^ '' array. The 0 temis indicatihg the solution are marked with aster- 
isks m the Cij' ' ^ anay. The corresponding terms in the Cij array are also 
marked with asterisks.' The sum of these terms is the solution sum, 
23 units, This may be checked by subtracting from 26, the lower right 
entry in the first array, the sum of the lower right entries in the two 

13 ' : 
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arrays beldw it. In more complex problems additional transformations 
of this type may be necessai^. But we can prove that it is ajways pos- 
sible to make transformations of this type so that the solution may, be 
based eventually on 0 elements* These transfoi^ations are fairly easily 
discovered in practice since they are based on pa^rns of zeros. 

The method is applied to a problem in the frequency form, previ- 
ously used by Votaw to illustrate the simplex method^ in Table VIIL 
The C|j^^^ array results from the subtraction from the maximum row 
values followed by the subtraction of the minimum column values, 
Butj considering frequencies^ there are not enough 0 tenns in row 2 
and row 3 of the third array to satisfy the frequencies. However, the 
subtraction of 1 from row 2 and row 3, with the subtraction of = 1 
from column 3 and column 4 (and a net sum of 30 + 35 = 25 ^ 28 — 
12) leads to the Cij^^^ array with very many O's. One of the many solu= 
tions immediately evident from this array is indicated by superscripts. 

A final illustration (Table IX) uses a modification of Brogden's 
quota form problem with 1^ men and 4 job categories. The solution 
follows the steps outlined above, Subtractions are made from the Cjj'-' 
so as to meet the quotas for column's and corresponding subti actions 
are made from the rows so as to keep one 0 term in each row. A 
corresponding treatment of thq first column of the Cn-^' matrix leads 
to the Cii^"^ matrix with enough 0 terms to reveal the solution indi- 
cated in the column headed J[. 0 

The solution sum 708 units can be obtained by adding the Oui values 
of the first array indicated by the solution. It citn he checlced by fonii- 
ing723- (6 + 9). ^ 

7. Concluding Remarks. The method " of transformations just de- 
scribed gives a solution to the classification problem which is as simple 
as one can expect. It can be programmed for machines but^ except 
for the most complex problems, hand methods are quite satisfactory. 

We now appear to have a ^ood solution for problems with true Cij ^ 
values. When tnie Cn vahieslare unavailable, as they commonly arc, 
questions arise as- to the estimation of the vahieSj as to the validity and 
sampling errors of the estimates, as to the resulting effect on the for- 
mulation of the problem, etc. The study of this general area, involving 
possible alternative procedures using the irtforrnalion available^ is very 
important. 

A narrower and more immediate practical problem also commonly 
confront us. Hfiw can we, with our present knowledge and informa- 
tion^ jg^lculate^hy useful estimates of the Cjj to which we can apply 
the Available techniquas? (See F, ) We might u^e standard scores 
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of variables correlated witfi success on th^Job, as in the Brogden 
illusttatioDj but, as the vocational counselor knows, such single predic- 
tors are not commonly immediately observable. And even combina' 
tions of observable predictors, such as those obtained by regression,* 
are not enough for this problem, even if valid; since they must be 
transfonned to estimates of the oonfribution to tiie common effort. 
Some sort of a weight must be given to each particular job since a 
measure of die importance of the Job, as well as the profldency of 
the individual on the job, is needed for estimating iie confribution of 
the individual to die common effort. How are we to determine these 
job wei^ts? Aside from the matter of the validity of the predictors 
we are forced, for the most part, to rely on the estimates of experte or 
to use hypotheti^l weights. In conclusion, I would like to pose this 
qu^tion for future research i How can we use available information 
in obtaining practical objective measures of those job weights whose 
determination is prerequisite to any useful solution of the personnel 
classification problem? 
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Table IIi Tlie Ge^ieral Problem 
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Table UIi Quota for Problem of Table I 





cii in $1000 units 
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Table IV; Quota Form for General Rfoblem 
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Table V; Frequency Form for 


General Problem 
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Tablet VIi Successive Trwsformations for Problem of Table I 
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Table Vll! Solution of the Problem of Table I 
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• - • ' ^ ; - * ^ - -U ' ■■ 
Tdble Sdution of Votaw Pabjein 
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^^abi© ym ( Coptinmd) 
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Tftbla D^s Solution of Brogden Prdblem \. ^ * 
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' Table IX {Cmtinued) ' • 
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Table K (Continued) ( * , , 
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New Light on Test Strategy 
Decision Theory* 



^ LEE J , CRON B ACH 



In every pracaeal use of teste, our aim is to make decisions. This 
; ■ is obvious in personnel selection and In Dr. Dwyer s assignment prob- 
Ian. but it is also true of testing in tiie classroom and in the clinic. LM 
teadier uses teste because he has to make decisions about appropnate 
instructional methods. The clinician uses tests as an aid m decidmg 
on tterapeutic tactics. Sometimes, as In vocational guidance, the de- 
oisions are made not by the tester but by tiie person tested. Test theory 

shoiid indicate hovv to reach the best possible decision in any ot these 

't. — - -situationsr , . , 

We use Ae word strategy to refer to the process by which an in= 
dividual arrives at a decision. A strategy may be very simpler I sha 
examine.the appUcant's grade average, and If It is B or better I shall 
accept him.-* The strategy may instead be complex, statmg what te 

if any will be given, what decision wiU be made for any particular 
pattern of resulte, and what furAer steps will be taken to decide on 
borderline cases. Choosing among alternative strategies is the essen= 
tial problem' of test theory. . .. 

There are two questions In choosing a strategy. First, with any given 
procedure for gathering Information, what is the best proced^ure for 
fcislatlng Ais Information into final decisions? Dr. Dwyer has just 
shown us the solution for one problem of tiiat type. The second, but 
logically prior, quesHoitfs: Among several alternative procedures for 
eathering information, which is most profitable? 

In order to compare, t%vo strategies, we have to detfrmme how 
much benefit we gain from either one. Most of the problems of de- 
cision tiieory therefore reduce to determining just how much benefit 
is eained from a parUcular decislon-maklng procedure. 

Since this morning's program Is Intended to deal primarily with 
insights from some of these newer points of view, I shall not dwell 
on Ae mathematics of decision problems. There Is available a large 
amount of relevant theory Itt the work of economists on ^J*^'^' *e 
Aeory of games, and In thfe statistical decision theory ^ Abraham 
on work conducted under Contract N6ori-0714e with the Office of 
Naval Besearoh, 
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Wald, itedsiori problmns mn be attacked in many ways, but we 
have mnBh^ ourielvei to steategies whidi maximize expected utility, 
Tt^ is tmsonMrn ^nly if we aire dealing with a stable and familiar 
sihiation^ " ^ 

f Hie.deoision model xequirei us to specify tfiree asptcts of any 
deoision. One is ft© proposed stoategy or decision rule* For example, 
rtie stoti^ m combine scores by a i^gres- 

sion foranula, and ao^pt everyone above a given cutoff « Second, we 
consider die ad^uacy of die iitfonnation to be used, Tlie usual Qon- 
ttagancy mateh or scatter diagram relating test scores and criterion 
ioores deals widi ftis question. Dr. McGiUs work with information 
analysis is primarily concemed witii studying validity maMces. Hie 
third n ^^ff^ ^ element is an evaluation matfix, ThiSj sometimes called 
a payoff matrix^ states sp^ifically just what beneflt or detriment ac- 
companies each ^ possible decision; Dr. D^er s Tables I and II are 
.evaluation matrices. (but his problem is so stated that the validity 
of the predictors also affects his entries). Once diese three aspects of 
a prdbrem have been described^ we' are ready to compute die payoflF a 
perison can expect if he bases decisions upon this information and this 
strategy. 

Dr. Dwyer properly drew attention to thm fact that it will be diffl= 
cult and at times impossible to write the evaluation matrix for a par- 
ticular sitaation. To let this difflculty deter us from using decision 
theoiyj however, would be to deny the possibility of sound test theory. 
Test effectiveness simply cannot be evaluated without aft evaluation 
mabix. Even the conventional procedures of test analysis assume 
certain payoffs coverdy, and the reasonableness of some of these hid« 
den assumptions ia open to question. In the future^ testers may wish 
to determine utilities by a sort of cost accounting in any specifle prac- 
tical situation^ in order to amve at the best decisions. Our project 
is proceeding along different lines. We are working wifli hypothetical 
(but we hope realistic) decision problems. By assuming that the evalu- 
ation mah-ix has some characteristic fonUp we are able to judge the 
utility of different types of tests and stategies. Such an approach can 
be no better than our assumptions. We hope nonafiieless to ^mve at 
ganeral principles of testing which will illuminate many real situations. 

Let me turn to some of tiie concepts a decision approach brings to 
our attention. I shall cover four such points. 

1, Our model suggests tiiat the vglue of test infonnation should be 
judged by how much it improves decisions over the best possible 
dedfi^i made wiAout the test, whereas the^nventlonal validity cd= 
efficient reports how much better test decisions are^han chahce de= 



30 



32 1954 INVITA^TIONAL CONFERENCE 

cistons. In Ae majoritjr of situftUons where teste wuM-|BTige^a— 
substantial amount df information Is dready available, and if no test 
were given tiie decision would stlU be considerably better ttian diance. 

Our most vaUd tests.are essentially work samples of &eir crifena. 
Where such a work sample might be used, evidence of past ^rfom. 
ance is also a vaUd basi^ for decisions, and such evidence u often 
readily available. In pwdlcttag school marks, for exMap e. a s^olasbc 
aptitude test Is not greatly more vaUd tiian past s^ool records. The 
conWbutloii of tills test to decision making is much smaUer than its 
zero-orfer vaUdlty ooefflclent would Indicate, because better-than^ 
chance decisions could be made without it, 

A similar conception applies to classroom testing,. The basic Imowl- 
edge and skiU obfeetives can be assessed wlfli considerable accuracy 
from day^to-day assignments^ a test can add only a smaU mcrement 
to flia soundness of decisions. On tlie other hand, a teacher has rattier 
little basis for judging wWch pupils have problems of adjustinent. The 
teacher may therefore gain more useful knowledge from a test ot 
adjushnent which has limited validity, than from^n flchifevement 
test which largely duplicates data already available. There are senous 
weaknesses In our tests for .such educational outcomes as creatmty, 
^ reasoning habits, attitudes, and appUcation of knowledge to problem 
situations. They are markedly inferior In validly to^tests of gen^d 
intelligence or factual knowledge. But the factors that make testmg 
difficult also prevent vaUd non-test decisions about these objectives. It 
may HierefOTe be wiser to use imperfect tests of important objectives 
that are hard to measure, than to use highly valid tests that merely 
supplement non-test' data. , j i- „ * 

mutllity analysis leads us to wamine the value of adaptmg to in- 
dHual differences In either selection or placement. This can best be 
considered in terms of a placement problem, such as assigning studmts 
to sections of freshman English accdrding to thely Initial abihty. We 
mlaht think of Ae various levels as predetermined, and ot the test 
as assigning persons to each category using Bxed cutting scores. It is 
sounder to see Ae test, the curricula, and tiie cuttlng scores as mter- 
locked. We can increase or decrease the demands mstruotion m any 
section m^es, to fit it to die abflity of the persons assigned. Und^r 
this procedure we benefit more from testing than when we leave the 

tjeataient fixed. . tt « 

Certain simple assumptions lead to interestmg conclusions. If a 
sample is divided Into groups, using fixed cutting scores, the extent to 
which treatment for ttie grouPS should be differentiated depends on 
the validity of die placement test. If the Information has zero validity. 
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u^fy is maximi^ whea we t#ac^ all s^oi^ to tib© mMner iuited 
to the avaxaga c^^a pdpi^tioD* As validly m^easaij ffig tr^taMts 
giv^ ttia ieetions may 6iBwt mom; but^no matter how vahd the t©s^ 
0i6re ii an optnaiua degree erf diffm^tiation of teeatment If Weat- 
meat ii differMtiat^ Mqrotid M point die benaflt from swUoning 
di^to^. IndMitt is jgossible to dUferratiate treatoien^ so radically ^ 
tiitt 1^ ta tra^^Wsidte from ieedoning ev«ar 4ou^ Ae t^t * used 
has considerable v^dity. 

^^ii anal)^^ raises si^mis queitipn as to whedier we are ri^t when 
we urge teachm to adapt to indlv^dud dffierences. If &e teaser has a 
stand^d plan, weU fitted to fta average of die group, he should hesi- 
tate to dqpart from it. Marked alteration of Ae plan to fit todivdduali 
appears to be advisable only when Individual differences are validly 
assessed and Aeir impli^tions f or treataient clear. 

3, We tun^ now, tS anofeer iug|^tion encountered in decision 
theory. It is customaty to look at a te^teas a imit and to use ^]ust 
- , tor one terminal deoi^ At .any point in testing, however, can 
make a tOTiinal decision or can continue to gaAer informati^. New 
frontiers open for us when we view testing as a multi-itage, cff sequen- 
tial, operation. / 

Suppoie, in a simple selection problem, we have several short apti- 
tude tests, whi^ together might constitute a selection battery. We give 
the first short test; some men cm be rejected or accepted at once, but 
less dear-cut cases are retained for fmther testing. iUFter ,the second 
test is given to these men, we can make more final decisions, and only 
a border^Qe group goes on to ttie third test Ihis process terminated 
when die benefit from information to be gained at any stage Is out- 
weighed by ihe cost of testing. Considering cost of toting, ^© se- 
- quential method is more profitable than giwig die same test to 
everyone; If testing is e^qpensive, one reaches the final decision for 
a surprisin^y large proportion of men, after only &e first short test. 
One paper on diis Une of attack has been pubUshed by Arbous and 
Sii^al (1) but oitf detailed resulte will differ from theirs in important 
^ respects. 

> A sequent ^an would require new ways of orgwizing testing. 
I shall diaouss one procedure for possible use^in vocational guidance, 
cUnio^ dia^osii, or evaluation ^f classroom learning, HereJt mii be 
described in terms of die job assignment problem/ For different jobs, 
so many abUltiea ^are relevant ftat we cannot hope to measure them 
. alLa^cu^^ brirf teste, however, one 

^uld OTidely measure^as many as vmables in a haH day. Such a 
suTO?y will indicate some jobs for which die man is m uriikely pros- 
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> ^ and a if eond group of jobs for wHich the tests show possible^ 

■ aptitii^^ for tte man's ^econd testsesiion, perhaps only an honr 
"^W* tiia.flrst test we would assemble^ a set of^ooklets to test Kim 

mor^ Aorou^y tiioie promising ap 
rovmg would be ebhtinued* When the fl 

w(md Have a|hi^Y^liable measure of ^e man's aptitude for that 
jo^ and also Vjood measurtf for flie ofter Jobs seriouslj^ considered 
, = as alternatives. But we would hav^ wasted little time in getting an ^ 
apcurate nieasura of his sp|ice ability or his dexterity, if these areas 
were not among his* better aptitudes on the fli^t sun^ey. 
We mi^t actually davelop different sorts of tests for the earlier 
- _ and later screens, The Strong Vocationd Interest Blank might be re- 
- ^ placed by a brief questionnaire, perhaps one page long. This seems . 
likely to identify the important interest groups for a given man. There 
might then be a separate, longer interest blank for each of these in- 
terest groups^ to provlS more pjecise differentiation between related 

■ CNDOupations timi is now poss^le^ ^ , ♦ 

4, Perhaps our most far^reachLng conclusion is that we should take 
a more favorable view of tests with low validity. Traditionally^ if a 
score has low validity^ we condude that it should not be used. But such 
tests become valuable when selection ratios are low (as Taylor and 
' Russell noted )V when they give even a little new evidence on an im-- 
' portant decision, and particularly when they are used as a preliminary 

^ survey, ; ,\ 

The survey is especially important when many decisions are to b 

made. Sometimes, as in vocational choice, the decisions are interrelated 
and lead to one final course of action. The decisions may, on the other 
* hand, be quite independent, as whan one diagnoses^many persons. 

The piSblem in testing is ordinaHly^terielect information-getting de^ 
vices \^ich will yield greatest benefit for the time available. If we 
spend a lot of time to get an accurate answer to one question, we must 
answer other questions without added information, In this situation it 
^ may be much wiser to use several tests of limited validity, so that every 

decisiori is made with some wisdom, than to get highly accurate in- ^ 
formation for just one of the decisions. _ ; / 

TWe difference between validity and utility is clear when we com- 
pare group and individual tests. An individual mental test measures 
one pereon, with essentially the same expenditure of effort by the 
. tester as the group test measuring one hundred. If the two tests have 
the same validity, the group test gi^es us 100 times as much informa- 
tion, and bears on 100 decisions while the individual test bears on one, 
Hence the improvement of decisions is vastly greater when the group 
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tot is used .If fli© indl^dual test has much Wper vdidity Aan tiie 
poup tes^ wUA is best to use wodd depend on tiie speeifle daeision 

This i^Miptioii p^mits a favorable vi^ of mt^rviewi and omer 
dtai^ ITOWlutBi whicA rover many aap^ of tfie pai^onaUty, ^ese 
meffiods, ftou^ undependablt, are suited to a wide^casting sur- 
p4^g a^ Utti© tafranaW^^ each of hunch^ queatiOTs* 
Sueh a pi^ltauiary swning draws attention to fte arltical areas v^here 
further tofonnation should be gadi^ed prior to any decision. The ta- 
ditionrt nanowly-foqiiised measuring deviea is ideal whra w© 
in advance ©xaedy wibfyt quastion needs to be aniwered* But m decid- 
ing whether a miin will make a godd axecutive, or in locating a pa- 
tim^s chitf fWifUrt a^p,^ no i focus is possible, ^e flnt task 
of ttio ass^or is to discover wh^eh arfti^ variables will dictate Ae 
"^per decislpn about tiie individual; in difforent cases, different vari- 
ably will be ariti^* 

. Personnel wdrkei^ regard the Interview as indispensable, and dini- 
daS have ron^^^^ metiiods and qualltetive 

analysis erf intelligence test protocols. In my opinion, this faitii hy 
develop^ l^galy bemuse of regarding experience wl& Aeae techf 
niques to tteir survey function, i.e.j as tiie first stage in a ieguential 
ais^sment. If It is tone Aat these multi'dimeniional teehnlques have 
a unique place to assessment| we should judge hoW well tfiay do diat 
job, and should not draiand tiiat they be good m^suring msbruments 
— whidii ttiey are not On to otor side of Ae pictuii^, if ttieto ptsper 
fimction is to make preliminaiy surveys so Aat morte totensive exam- 
ination cm follow, one should not rest final judpienfa on tiiese fallible • 

'instruments. ' ■ [ ' 

' Taken as a wfiola, decision theoty is a maditoatical system which 
permits us to examiiie ttie problems we face in davelopmg testi, chpos- 
ing between teste, and iriteipretirig tests. Whenever we ^ specify 
any particular decision problem to the detail ftofessor Dwyart prob- 
lem required, tiien decision iheoty can tell us just^what to do. By 
studytog common type-problems, decision tiieory can also offer gen- 
era! recommendations regarding testing strategy/^ 

Conventional test tfieory assumes that we use tests to obtain nu- 
merical measures on an Interval scale, as in |&e physicd sciences. TOat 
is rarely or never true. The fbnction of psychological and educational 
t^te is to aid m mddng discrete decisions. TTie greatest contribution 
of decision Aeory Is to help testers see tiiis function more clearly. 
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The Relation Between Uncertainty 
and Variance 



WILLIAM J. McGILl,. 



ITiere has bean a peat deal of hue and ciy about mfonnation Ae- 
^oiy in ^y^olopcM circle. In tihe midst of Ais hue and ery, it is easy 
to become confused about where information Aeory is suppwed to 
mdce contact wi& p^chologyp What^ if anytiiingj should infon^ation 
tiietMfy COTtfibute t© psychology? ^ 

The theory deals witii tansmitting symbols from one system to an- 
odier* Thftt certainly sounds familiar. Many questions th^t come up in 
learnings in perception^ psycho-physics, testing (just to mention a 
few areas take on new sipuflfiance in tiie terms of information theory. 
ConsequenUy when we read presentations of inforrnation ^eory and 
run awoss words Iflce "messagej*' "noisCj'' '"channel capacity" our nos- 
trils begin to quiver and we sniff a familiar scent in tiia air. At once 
we try to find analogues for die classical problems of psychology in 
the Aeorems presented to us by tfie communication engineers, "niis 
is one large zone of contact between inforaiation &eory and psy- 
chologyi Informatioa theory is a source of analogies and ideas ttiat 
mi^t not have occurred to us, if we tiiought about our problems in 
anotiier way* Perhaps the analogies are helpful, perhaps not, I would 
radier not discuss the merits of using information tibeory for tiiis pur- « 
pose. I bring it up only because I want to' hammer at the distinction 
between information theory and information measures. The tiieory 
is concenied with bransmitting symbols despite noise. Information 
measures are concerned with the arithmetic of mean-log-probability. 

Mean-log-probabiUty is something lik^ variance. It measures die 
amount of spread in a discrete probabilflF disWbution, The formula 
for mean-log-probabiUty is usually written as follows: 

. l/(y) - - S pCk)lDg,p(k) 
^ k-1 
In this fprnaulaj y is a variable that can assume any one of r discrete 
values. Each of these values has some probability, p(k), ITie negative 
sign brfore tiie^ summation insures that Xr( y) is positive. The interest* 
ing tting abqut mean-log-probability is tfiat any numbers or meas» 
UTOTaents which the various categories of y mi^t represent do not 
appear in the fonnula fojf U(y). In other words U(y) is non-metric, 

, 36 . 
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Mean^log.probabUi^^ftan be calculated for a vaJiable like "methods 
of parental discipline" (just to choose an example), where, no num^ 
bars can le attached to any of the mettiods. One method is as good 
or as bad as another. All you can say about them is Aat they^are dif- 
.fwent. Obviously, you caVi also compute mean-log-probability for 
meWo variables, like IQ, but in so doing you sacrifice whatever you 
iBight gain from knowing Aat an IQ of 100 is viry close to an IQ of , 
ipS and very far froni an IQ of 150, » 

In die formula for mean-lpg-probabiUty, logs are taken to tiie base 
2. Tliis is done in order to provide a simple unit— the "bit." When U (y ) 
is measured in bitsj It turns out that U(y) is the average number of. 
binary, or two-category decisions required in order to identify one of 
ttife values of y exactly. It is not really necessary to measure mean- 
log-probability in bits, but almost everyone does. 

You can apply meanJog-probability in many situations where m- 
formation theory has nothing whatever to say, and where in fact an 
attempt to apply infonnation theory might even look a little ridiculous. 

For example, suppose you want to express the relation between 
anxiety level in children and several methods of parental discipline. 
The relation might be demonstrated very easily by using mean-lbg- 
probability. But our understanding of the relation would not be en- 
hanced if we considered the disciplinary metho^s^as messages and the 
anxiety levels as the versions of these messages received at the end of 
the channel. The "capacity" of such a "channel" is of Uttle interest to 
us. The fact that we can with effort construct information-aieory-type _ 
interpretations of rejations like this one, merely demonstrates that 
communication engfueers and psychologists measiwe things with 
roughly the same kind of arithmetic. It does not mean Aat feifonnation 
theory-h&s anything significant to contribute to our understanding of 
the relation between disciplihe and anxiety. Consequently we can be 
interested" in the information measures entirely apart from their sig- 
' nificance in information theory. To fortify this distinction, Jet us now 
call diese information measures by the n^me "uncertainty" measures. 
Uncertainty means mean-log-probability. It has no necessary connec- 
tion witii information theori/. 

' In this paper I want to show that uncertainty Is essentially non- 
metric variance. This can be a very hollow claim revealing ilttle more 
than a superficial similarity unless we are prepared to outline in de- 
. tail the properties of variance that uncertainty possesses. 

What are the principal properties of variance? 

1 Variance is a measure of spread or variability— so is uncertainty. 

2. Variances &at are independent are additive— uncertainties that 
are mdependent are .plso additive. 
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3, Vwkne^ can ba ^irtltiQ&ed into compotfMts Aat reflect cdntei- * 
butions to fte totol varimea froBi a number of predicting variables ^ 
takm one at a timfe and in combfaiation; uncertain^ has precisely tiie 
saml property. ' 

4, Variances are seriously affected by ^e metric or scale propertias ^ 
of fte data from which Aey are derived* On the other hand uncertainty 

Is imiAected by tiie metric or scale properties of the data. You can 
chop a diBMbution into ^rts and reMrange all tfie parte wi&out 
. changing Btoqerteinty — ^but the variance will change radically. 

It is^ obwous fliat uncertainty and variance are closely related but 
it is 'equally obvious that the two measures are not identical* There is 
no simple equation that takes you from uncertainty to variance. In- 
stead, the parallel is in tenni of sbructure or operations* To make tiiis 
point clearer I have prepared two tables. Tables 1 and lA in the 
handout. Table 1 shows tiie symbols^ formula^ and definitions used in 
' a double-classification analysis of variance* Table lA shows flie same 
: airangOTent for a double-classiflcation uncertainty-analysis. You can 
see by glancing across the tables Aat the parallelism is veiy complete. 
In both^cases the predictor variables do not have to be metric* They 
^ can be pure classifications like our example of methods of parental 
disclpUne* In ^Variance analysis the dependent variable is metric. 
It is something that can be measured numerically, I have pictmred the 
dependent variable also as having discrete classifications but this is 
just a convenience of notation, 
Ypu can see the relation between the two analyses very clearly when 
- you look at this pair of equations: , 

;U(y)=U,.(y)^U(yiw,x) (1) 
' V(y) = Vw«(y) +V(yw,x) (la) 

The definitions of the tenns in these equations are explained in Tables 
' 1 and lA* The equations state that die, variability of th^priterion or 
depen^nt variable can be analyzed into predictable and'^predict- 
able parts* Furthermore the predictable variability may be decom- 
posed df follows i . ^ 
' U(yiw,x) = U(y:w) + U(y:x) + U(yj^) (2) - 
V(ytw,x) ^ V(y*w) + V(yix) + Vty*wx) (2a) 

where again U stands for uncertainty and V stands for v|iriance* These 
equations state ttiat ttie total predictable variabiUty can be broken 
down into a part predictable from w, a/part predietable'from x,"and a 
* p£rt ^edictaMe frOTi uni " 
Spmetfaes the uncertainty interaction tenn, U(yiwx), can be nega- 
1 ' tive. What istnot generally realised is that T(y*wx), a component of, , 
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variance^ can alio be negative if it is defined as it is in Table !♦ The 
reason is tiiat this term contains a hidden cross product a correlation ^ 
tenn, ap well as a sum M squares. Furthermore the correlation term 
has a negative sign and can sometinflSs be greater in magnitude than 
the interaction sum of squares. This means that the mteraction term 
in tile variwiee-analysis <^n sometimes be pushed negative by the 
negative correlation term, ^ 

When does this happenP It happens when Ae predictor variables ^ 
ar§ not ^independent In ordlnaiy analysis ot variance Ais situation ^ 
never comes up because we are careful to consti^ct tiie analysis so 
that tiiere are equal numbers of observations in every celL/niis ayto- . 
matically insures tiiat the predictor variables are independent. We ca^ 
guarantee tiiat things behave properly in botti uncertainty-analysis 
and variance-analysis by making the predicting variables ortfiogonalj 
i.e, inde pendent. When the^redictbr variables have lome soft of 
.mefi^ this is equivalent to tequirin| Hthat die correlation between the 
predictors is zero. In twt,^ all tiie arguments I have been maldng for 
the* similarity between uncertainty-analysis and analysis of mriance 
are equally tarue of naultiple regression analysis^ i.e, the case B which 
the variances in Table 1 are computed around a regression plane. In 
multiple regression any interaction computed ^ the rule shown in 
the last line of Table 1 is bound to be due to non-orthogonality, .^^^ | 

We sde that multiple congelation, analysis of variance an4>uncer- ' A.**^ 



tainty analysis are aU very closely related. The different pj^dictive 
methods are necessitafed because^ sometimes the predicting Variables 
have metrics and sometimes nots'^ spmetimes the dependent variable 
has a metric and sometimes iiot ^ 

What remains stable in each pniB is the stmc%ure of tiie statistical 
process of prediction, the operations in^falred /ip making the pre- 
diction, K ^ ' ;^ 

One important ^practical consequence of t|iis ..'invariance of struc- 
ture in statistical prediction is that we can analyze non-metiric 
data in the manner of analysis of variance without resorting to the dis- 
tortion of giving the data an artificial metric. The non-metric analysis 

: iV qarried out with uncertaifity measures mA meanJog-probability, 
i^Tiother ve^* important consequence is that fill the literature on ex- 
j^rimen^al design in variance analysis is directly applicable to^the un- 

' certainty toalysis, r 
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% PARTICIPANTS 

LfcE J, Cronbach, Edward E. Cureton, Paul S. Dwyer, Dorothea Ewers, 
M. M, KoiTic^ WitLiAM J. McGiLi^ VicTon H, NoL^^ 

Dr. Nou-r This is by way of comment rather than criticism on a 
pomt that Dr, Cronbach made in his paper, I have reference to the 
second point in relation to use of tests in providing for individual dif= 
ferences, . 

It seems to me that tiie implication tiiere was a little unrealistic 
in that we do not ordinarily, in the classroom, attempt to provide for 
individual differences on such a mechanical basis. We do not say, 
"Your IQ is 117; consequently, you do this/' and 'Tour IQ is 90, and 
you do tiiis." 

It seems to me it is more a matter of .providing a variety of materials 
from which the- student tfien, by a process of selection, determines 
what is suitable lor him. In odier words, it is more a cunicular prob- 
lem than it is a problem of measurement. Perhaps I am putting an im- 
plication on Dr. Crqnbach s statement which he did not intend, but 
I think it is important diat we think of it in terms of providipg a va- 
riety of experiences from which students pf dififerent types of ability 
and interest tiien choose rather than a mechanical process of deter= 
mining, on the basis of a test. Just what a student is going to do. 

Dr, Cronbach! I agree with Dr. Noll's views on the curriculum. 
The generalization of the paper is tiiis: our system of analysis forces 
us to reconsider much of the docbrine we have had regarding indi- 
vidual difference. This doctrine takes a variety of forms. For instance, 
experiments on homogeneous^^gtoqgfs have not le^ to conclusive results. 
In these studies, pupils have been given different treatments, beat- 
ments varied rather mechanically, plus some additional flexibilities 
such as Dr. Noll mentlpTied, The results to be expected from X=Y-Z 
sectioning will differ pfea^Uy, depending upon the extent to which X 
pupils truly differ from .Vs and Ys differ from Ts. 1 know of nothing 
done in these research programs to make sure that the adaptation to 
individual differences applied in each section was optimal. 
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M teatawt is. oyerdiffarentta^ or underdiffarentiated, Aey cer- 
tainly do not get 4© full value of homogeneous grouping. 

Our result also bea^ on Ae assessment literatitfe. Thus it has been 
s^aid that tiiertfmust be something wrong with counsel's because Aetf 
standard deviation of estimates of ability Js lower tiian the measured 
standard deviation. But die evidence is that counselors are making 
mistdces chiefly because they are ou^fdifferentiating, taking tiie tests 

too seriously, i. j • 

Obviously we have been encouraging teachers to look at the students 
personaHtici and to handle some of: diem. differently from ofliers, on 
basij oHud^ent tiiat one i^ds encouragement and another needs 

stern treatment . ^ 

The answer lies in \hm direction: it is fine to differentiate ^eatment, 
so long as you do it on a tentative basis, and allow for Wal and error. 
If you make irreversible decisions, such as you cannot go to college/' 
then your information has to be extremely good, 

Dr. CuroroN: It seems to' me we might consider one illusttation 
while we are on Als topic, because it happens to be fairly common in 
Americ« universities and to rest on pretty definite information of a 
non-mettic type. Tliis Is the matter of fifeshman Eriglish, I tiiink prob- 
ably in opUeges and universities you will find more specific sectioning 
in Englisli tiian in almost any otiier area. The reason is that tiiere seem 
to be some fairly definite cutoffs which imply specifically different 
types of insfruction. For example, if a student can write good English 
sentences quite regularly, and can also handle them in parapaphs and 
larger units wifli acceptable style, many universities will excuse Aem 
from freshman English at the outset. Secondly, if he can write agcept^ 
able grammatical sentences, but does not db too well in organizing 
tiiem into paragraphs and larger units and produces poor style, he is 
likely to be assigned to a regular section. And finally, if he is unable 
to write grammatical sentences and handle properly the basic elements 
<of grammatical structure, then he is likely to be assigned to a special 
section in English in which he will get the type of drill that is not 
needed by tiiose whose secondary preparation is somewhat better, 

Dr, CBONBAOr: Our analysis suggests research on the performance 
of students of differing initial ability under various treataients along 
this eontinuimi, from very routine English training to the highest level 
of English, The highest level might involve going into matters of style, 
for instance, well above the usual considerations of clarity. 

Dr, Kmnmi This is for Dr. Cronbach: I was wondering about us- 
ing the sequential method in entrance examinations. 
I carried out a study of the effectiveness of enta-ance examinations in 
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our M^ac^usetb State Taa^ai^ CoUegef , W@ found Aat by the end 
of ioplioroore y of tboi© people who were admitted by certifl* 
^tion, only 6% '"flunked ouf* or dropped out with low grad^. We also 
found that for tixoie people who were admitted wi A an entrance exam- 
ination mBik of over 160^ only 18% were dropped^ Of coiLrie, tihie mark 
of IW or any oAer mark doesn t have meaning unless you know what 
it standi for. We. used a ^mbination of th© score on the psychological 
examination and ^e score on the English Examination^ 

"^^^t I am particularly interested is Ae low^t group. Wa have 
a range between 130 and 140 which is questionable. From itmt group, 
48% ^flunked out/* ' 

I was wondering if 4t would be a good suggestion to see whetiber we 
could iise soma sort of sequential testing of these people^ initead of 
going to the expanse of putting &em throu^ the freshman or sopho^ 
more year and then having practically 50% of them drop out. 

Da* CEQNBAcmi In sequential tasting you often can gat more thor- 
ough measures than you Muld l^rd to give to averybne. Tliare is no 
compeUing reason why borderline cases could not sit for a bvo-day 
examination #ven thou^ you wouldn't think of doing tbU with eveiy 
entrant. However^ you are likely to make a great deal more gain if you 
find soma new sources of teste that will supplement your prediction 
rather than just extending the old one. 

The statement tiiat was made focuses on die importance of hawng^ 
very clearly in mind your value assumptions before you hy to make' 
decisions about test sfrategy. We can each make our own value assump^ 
tion* I would be hesitant in tiiis particular example to accept the impli- 
cation Aat if a person flunks out after two yeai^, tiiis result has cega- 
Itiva utility. Hiera may be positive utiliiy in those two ^ears of callege, 
particularly when Judged from the viewpoint of the pdividualj rather 
Aan from the values of die institution* I ttdnk we fall too easily into 
seeing the problem only from the institutional side* / 

Dr* Ewms* I simply want to add to Dr. Cureton's statement that we 
do the same thing for freshmen in high, school, except we have one 
mora category, ttiat is, if they cannot read, we put them in a fourth 
group. But I wonder really how well we do teach them in these various 
levels. If we could do a better job there, we might not have so much 
trouble at the college level 

Dr. ZuBiNi Dr, McGill, I am a little 'puzzled by your analysis, espe- 
cially by die terms you used. They evoke memories of pravious faiowl- 
edge and I wonder whetiier you would like to connect up for us die 
previous meaning attached to these times witti die meaning you have 
given Aem. Analysis of variance gives the component of the total 
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variance in a systematic fashion, and, of oourse, leaves a Utde bit for 
the uncontrolled variance as an error, term. Ordinarily we refer to it as 
the standard error, somettilng left with which to measure the signiii^ 
cance of the single components. As a matter of fact, what w rela= 
^tionship between uncbrtainty and \i)0e older ideas, of standard errorj 
Are they at all related, or are they two completely independent thmgsC 
Dr Dwyer's method, of course, applies only to situations where you 
already know the value of each Job and you know the value of the 
person, and so on. Now, could that method be appUf d to situations 
wKere we still d& not know these values? In the clinical field, for exam- 
pie we do not know the amount of money we could save by a certain 
procedure. Would It be possible to set up a contingency table with 
unknown c., and then solve for them under certain conditions^ Rather 
than assume you know your Ci,,, could you put in specifled cj., and 

see what happeris? i u 

Ete McGni: The answer Is that It was my intention to try to evolce 
associations from as many people as possible, because I believe that if 
you have the old associations firmly in hand, it is extremely easy to 
manipulate these so-called new concepts. They are not new at all. 

The analogy of uncertainty with error variance is an example, the 
, "error" uncertainty contains error plus everything else you forgot to 
analyze. If you have constructed an experiment properly (which almost 
nobody ever does), the error will be what the model claims it is. 

Theilnterestlng thing kbout uncertainty analysis is its technique tor 
testing null hypotheses.an this analysis you do not test the predicteble 
components against error; you test the error against zero and the pre- 
dictable dbmponents against zero. It is a tricky little switch, bu the 
Interpretation of the components Is Identical with variance analysis. 

Dr Dwykr; You can't work the problem unless you have some 
lvalues opo. . There isn't any reason why you can't have a whole series . 
of hypothetical o,j, and maybe the collective problem would be some 
sort of answer. This formulation of the problem demands it, but we 
could have a whole series of hypothetical values, which might be 
interesting. 
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Some Recent Results in Latent 
Structure Analysis 



T. W. ANDERSON 



Latent sfructure analysis may be thought of as an analysis of dis- 
crete data that is analogous to factor analysis of continuous data. The 
usual model for factor analysis is 



SI 

(X> xt ^ 2) M. + /ii + Ui, i ^ 1, 



»=1 



where Xi is &e i-ft test score, f.* is the ^-th factor score, is a con- 
stant and uj is the sum of tfie Uth specific factor and Ae eiror of meas- 
urement, The test scores are observed; die quantities on tfie right hand 
side are unobserved or latent 

For convenience of discussion, we shall assume that the fi. and Ui 
are normally distributed. The key assumptions of this model are that 
tixB sat Ui are statistically independent of ( uncorrelated witii) the set 
fv and tiie U| are statistically mutually independent (mutually uncor- 
related), Hiis means tfiat if we take the subpopulation defined by tiie 
requirement that the factor scores are given values, tiien in this sub- 
population &e test scores are statistically independent; that is, given 
the factor scores th& predictability of one test score from anoaier is 
zero. Mora specifically, given the values of tv, the test scores are nor^ 
mally and independentiy distributed with means SyAii.fi, + ^t, and 
variance These assumptions are reflected in die formulas for tibe 
variances and covariances of the test scores. If we assume the factors 
are uncorrelated and have unit variances and let the mean of Uj be 
zero and tte variance of U| be E uj^ = ar, then the variance of the M 
test score is 

(2) E(xi ^ m)' - 1 Xi.^ + 

and tiie covariance of two test scores is 

s* 

/ \ 'Hi ^ 

w ' E(xi - /ii) (xj - Mi) ^ S Xi, Xj,, i ^ j. 
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We can say that the common factors "explain" the interdependence or 
correlation of tiie test scores; the effec'l of the u. appear only m the 



variances. 



In considering latent structure analysis, we shall assume that each 
item is a dichotomy; that is, an item score takes on the value one for 
a positive response and zero.for a negative response. Thus the con= 
tinuous test scores x. are replaced by the discrete item scores y,. The 
continuous factor scores fv are replaced by the latent attributes g., 
which may be discrete or continuous. In the subpopulation defined by 
given latent attribute kores, the responses to the p items are statis^ 
tically independent; in other words, given the latent attrlbi^e score 
the predictability of one Item from another is zero. In such a 5ub= 
population let w,(g) be the probability of a positive response to the 
tth item- for ease of exposition wc shall assume there is only one latent 
attribute score g. Then, for example, the probability of a positive 
response on every Item is v ♦ 



(4) 



Pr . yi £ l,y2= 1, . . . ,y,. - 1 ! i J - 



iff2(g) . . . fff. (R). 



I 

For given g, the y, are a set of independent binomial variables. Next 
wo assume that there Is a distribution of the latent attribute, s.-^ f (g) 
over the whole population. The probabilities of y„ , , . ,yp or the rela- 
tive frequencies of variou.f response patterns for the entire population 
are obtained from those for the subpopulations by averaging with 

respect' to f (s)- . , , 

To make these ideas more concrete we shall consider two specia^ 
cases of the latent .structure model One of these can be derived 
from the factor analysis mWel, For simplicity wo shall assume that 
there is one common f^^^ ^ ""^ ' ^ ^' 

(5) -f^ ' ^hi = U + Ui, , i - 1, . . . ,p. 

Suppose that the caM&^ score x, Is replaced by a dlc^iotomous var= 
iable y. with y. =^lHirfi i^a. and y. - 0 if x, < a. From .(5) we 
know that X, .. are normally distributed with means zero, van- 
ances X,^ + <f,i and'covj^ari(^GS A, A,, and from this fact we can com^ 
pute the probability; of^^iftern of the dlchotomous responses. For 
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V/hmm h (xi , * , * , Xp) is tiia djenii^ of the nonnal distribution. When 
{ is fi^ed, tibe conditional distribution of Xi is noimal with mean Ajf and. 
variance ofi^; in ^is subpopulation the probabiUty of a positive re- 
sponse on the i'tik item is % 



(7) Pr{ yi ^ 1 If 1 - n (f) - 




where # (b) is Ae probability of a unit nomial variable betag less 
' dian b* From tfie fact that the ui are statistically independent it fol- 
lows that in the. subpopulatiop of a given f the Xj are independent 
and therefore the yj are independent For example^ / 

(S) Pr { yi - 1, . , , , yp ^ J I f } - ri(f) . . . wM - 

h*(^')]-h*/C-^')]' 

^ The case of one latent attribute (that is^ one factor score) has been 
of particular interest to sociologists and social psy^ologlsts. The itmns 
may have to do with opinions on various questionSj all of which are 
related to a given attiRiHe, For instance, the items may be questions 
such as '"Would you like to work with a negrof* *Would you like to 
lite in a nonsep^egated community?*^ etc. The underlying attitude is 
tiiat of racial prejudice. The end puipose of the investigation may be 
to rate tiie respondents on a scale of racial prejudice, 

In (7) fri(f ) is given as a particular function of f. Other functions 
may be postulated* Lazarsfeld has been studying matiiematical and 
statistical problems ftat arise when #i(f) = ai + bif + Cif^ or iri(f) 
^ai +b|f^» (0<f < 1)* 

Perhaps die simplest of ,the latent structure models arises when 
tiie latent attribute is discrete, guppose that the attribute can have 
one of q values, say f ^ g. ^en fPi(f ) can be designated as iri% 
where*a ^ 1,, . q, TOe distribution of die latent atti-ibute is specifled 
by Pr ^ f " ff Vs. The probability of diawing an individual from the 
«-th latent class is vsi the probability of drawing such an individual and 
getting a positive response on tiie i-th item is\v^i^i and the probability 
of drawing some indiyidual and getting a positive response on the i-tii 
item is . 
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(9) *i 



Staflarly, Ae over^aU probability of getting positive responses on 
items i and J is 



(10) iru ^ S ir^w^P^, " ^ 1 * 

t 

and the probability of getting positive responses on i, j, and k Is 

(11) inik = i iri- iri« ffk- p„ i ^ j, i k, i ^ k. 



There are similar expressions for the probabilities of other sets of 
responses. 

The wu mi, ana are known or are estimated frofti die datai the 
m- and V. are to be inferred. When the structure is known a respon- 
dent can be dasslfled into one of the q latent classes on the basis of his 
responses to tiie Items. 

A method for inferring wi« and v„ suggested by Bert Green [2], 
is based on factor analysis methods. Equation (lO)^can be written . 



(12) wn^ ^ (^i* ^ v^) i^f ' 

^ ■ -I 



these are identical with (3) if A„ = w,» V v„. The^m^toix (w.j ) can be 
foSored to determine the matrix (X„) = (if,«Vv«), but there is 
left^flio indeterminacy of rotation. To eliriinate this. Green suggests 
dso factoring %,m,^. Generally tiiere will be only one solution that 
factors both mattlcesi in the case of fallible or statistical data there 
wiU'be one factor matrix that gives the best At. (This descnption of the 
metfiod should be taken only as approximate.) 

An important difficulty with this method is that m „ wui, etc. are not 
deBned and not observed and these must be approximated m some 
fasUon; tills causes particular trouble with the second matox _(S,^,j.) 
aiat is factored. Anodier meAod [1] which bypasses this difficulty as 
weU as the need for factoring large matrices is to use only part of equa= 
tions (9). (10). and (11). The part of (10) that is ^ 's a set thrt j 
entirely below Ae mam diagonal of the matrix (w.,). In tiuj meAod 
2q« of tiie observed m, mi and mi, are used to infer the swae number 
of Ae latent parameters. THe algebra that Is Involved is the standard 
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theoiy of charaGterijtid roots and vectors, but we cannot go into details 
here. Tim mettod has been developed further to «se mora of tiie data, 
and a large sample tiieory of teste and confidence regions has been d^ 
rived* Neither of tte meAods mentioned is in principle efficient^ 
using all of iftfomiation in tte data, but eitiher onemakes it possible 
to analyse data in ferms of tiiis model However, further developments 
in statistic^ inference are needed even for the simple case, 
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The "Modliti;^ariablB^V as a Usefur , 
Tool m prediction , . ; 



DAVID R. SAUNDERS 



• This paper is intended to be partiy Infonnative and partly pe^ 
sua^e. ttie mfonnation standpoint. I hope to supply you 
ftn^wL to fliree main questions i What is a moderator variable? How 
S^;^,^t^ratoJ variable? and. thWl^Hy^sl^dy^^; 
.,V.ay? The persuasivp aspect of tfie paper is pbVious m 
S just afta the is an important aspect. because.mu^ dj 

formation I can give you is neither very new -^^^^ W --Pl-^| 
Moderator variables have been used'for many years by ^^f^^^^ 
iA ewiSmi^ and agriculture to help in fitting regression surface to 
-S^Sl Aese%pUcations 4he moderator effect is typically ju^ 
^^fmany tiiat are lisible V^th* multivariate curyihnear regre^ion 
a poUiS explslon. Our economic ^d agnciUturd 
fwfnl don't need and dont seem to have any sppcial name for it. 
^ WoS^^SnSiiight be tempted to -^^^^^5^ 
Sdotl^W^Ut this is a term that already has a lot of additional 
tientiflc comotattons that we want to remam neutral on. ^ 

lolar krL.; Gaylord and Carroll were pBrst to use a moder- 
at^^Ie in ps^chigy. They ^^^^^^ 
able." and presented-a paper on it c^rmg the 1^48 me^|B J. 1 tie 
term "population control variable" if a good one. because it sugg^ts 
a ^rv taportant application. But it is a bad term to the extent Aat it 
teSrUh^vp to a number of other equally important apphcations. 
whSi I wmtolh on. The term "moderator variable" seems to be gen- 
^;1ni::gh!n its meaning, and still not to be loaded with too many 

"'tiJ!^'?^'^^~d this thing, but have only -hmt^^ 
w^rt irtefs look at^some examples. By now I've managed to 
^ do^^ of attractive hypoflietical examples, but I'll W^^J^^^ 
and emphasise two examples that have been fully worked out and 



to ^mple is the one that originally led me to think amodj. 
atofvartbirmight be an important concept. Frederiksen and MdviUe 
a Em had just shown that interests were less predictive of academ c 
sic^ss for "compulsive" people than for "non^compulsive people (2). 
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For t|fa discussion, we can-igndrl tiic Aeasufe^'^^ to de-^ 

Bna Interaite, success, and tomgujsiveness, wcept to note that they all 
were regarded as continiiQUs vkfiables. Frederiksen'ff ftnd Melville's 
ei^^dmental dwgni rieVfertheless, had to be tiie analysis of covariance- 
the total experimental sample was, arbitrarily dividecl near tli© median, 
compulsiveness score to pioduce two groups— called "compuliivai''. ^ 
and "non^compulsives.'' The relations be^een interest and success / 
were compared for these groups, and were found to differ signifl&tttiy 
in the slopes of the regression lines, This was don^ separately for ten/ 
different interests as predictors. . . , V 

This example is typical of rWtny situations in which the use of a . 
moderator variable should be Considered.' this partitular kinH of ^ 
situation the term "population control ypri&ble" would also be apt. 
Clearly, what .we are after is a means of treating compulsiveness as thd 
continuous variable which it Is— a means of '^voiding the arbitrariness 
of dichotomizing or otherwise dividing the population into smallet 
pieces— a means of maintaining the integrity of the total population 
while stiU maintaining a statistical control on each individuaFs mem- 
barsbip m one of a continuous infinite series of ^sub-populatibns de= 
fined by his compulsiveness score. In short; we will allow the meaning 
orhis interest score to be * moderated" by his Hpompulsiveness scpre, 
r-^^lijs^^^ this turns out to be extreinel^ simple to do. Suppose 

I Sve/start )WtB an ordinary Jinear regression usin^ s^veral. y^&riabte/ 
. Keepmg eyfeKKd^^g scores^ we would write tKe equation 

' r ^ ^ SiKu ■ 

where y is our criterion, and pi is the beta-weight for predictor Xj, Now 
suppose that ^j, instead of being a constant, is itself a linear functlot 
of a series of moderator variables, z/s. If we plug this into our equi 
tion, and do a little re^arranglng, we immediately find that we can write 

k ^ ■ 1 , ^ , ^ ' ■ ■ . ■ 

This would be just another linear regr^si'oh if It were not for the l^^st 
term, involviilg tiie products of the x's^^nd ;the zs, If we want to, we 
teSi* alvvays choose origins of measureniehtfor Tthe x's and z's that will 

>rn0e everything drop out of this equatfrj^n except the products, and a 

"Constant tenn. 

It is evidently these product terms whlc|i are inextricably tied up 
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with the use^ot'a Aler^or variable. And that Is all that has happ^ed. ■ ■ 
So long as lhemisi blear separation of the x's and z's, we cannot have- 
any use for sqUired variables, let alone terms of higher power. All we 
have to dd to* 4hemodel to data is to and the appropriate xz product 
(or products) for^ch- subject, and treat It (or them) as new mdc- 
p^dent^reiiitors'to iny standard multiple correlation technique Of 
.O^e:^vJ:.civwwt. induce a product variable into a battery unless 
^bdth'of its factors ate already there. , i . 

' There are many mterestlng mnthematioal sidelights to this thing, but 
• if. 1 want to persuade you that moderator variables are useful and 
ptacticalrwe'd better 'go back to our examples, , , j 

W& took ttie data that Frederiksen and Melville, had c-ollected, and^. 
compute'd die products of the interest predictors with compulsiveness 
for each individual subject. Then we ran multiple correlations, ftrst 
adding compulsiveness to the Interest predictor, and then adding their 
product to the battery (3). ' 
. For three- bf the ten interest predictors studied, the simple addition 
of compulsiveness. to the battery gjive a- significant increase to the 
multiple R. In these three cases the compti Is iveness, score hapi^ned 
to act as a suppressor variable. A moderator variable is diffeyent from 
a suppressor variable, though they both typically have zero zero^order 
relation to the criterion- a moderator variable does not have to have 
zero-order correlations wi til, *ven the predictor. , 
. Back to the example: In five of the remaining seven instances, addi- 

tion of the appropriate prpdlict score to each battery of t^^ measures , 
' resulted in a further significant increase in the multip e R sign ; 
' of the beta weight for the third term was correctly predicted from the 
; iiypothesls in' all"t6n„.of .the ten instances. You will recall that Hie 
^^.H^pbt^esls told us a^^ end of the compulsiveness scale to look for 

;ff6od predictions » ^ ^ ^ s ^ r 

' ' .Thes^ results looked promising. So we moved ^tlitf scene of operations 
from Princeton to Rochester; from; (1 'group of self-referred counselees 
to a larger grc^up tested routinely diiriiig Freshman Week; from Strong 
Interest Blanks scored with weighted responses to scoring with unit 
weights The criterion was still freshman grade average for engineer- 
Ing majors, and the moderatpr>as:still the Accountant Interest score 
of the Strong, as timeiisura of compulsiveness. Insofar as the sameor 
■ similar interest scores; were available to try as predictors, wo were able 
' to cross-viiidkte all but one of tlic statistically significant findings from 

Princeton. ^ : . _ » - ; 

Tn this example we have observed that in Uie more signiBcant 
. stances, the predictive conh-ibution of the moderator effect is- just as 
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great jls dia kdtial oontiribbtion of A(i 'interait. variable pradictort 
TUs Tmyynot seem like skying mud, but note ^at Fradaril^en md^ 
^;;^M€ih4U# I^Ort eorrtlatipns rangirig all the ^$Ly hom zero to ovar 
flity fen* su^grouj^^ psing a pair of 

^to a tj g lidriveaas meiE^ ^ Eym^^v^TOTa modef ttl ad lapa^ston model 
do^ not l^d to madi in^ease in the multiple E» it dc^ lead to quite 
dffi^rat predictions for iom^ indi^dualt, aq^^ould be used if its 
effeet is eyen statiati^y si^U^nt. "^ess prediotions may diff^ in 
i^daid ^nror as well in die expected value ibe^» a 

In tiba f^nd main example tiiat I want^o discuss bare, ni^Atf tiia 
mq^p0^^ j^ pH^i^^ bas. baen^^omi to bava a significrat zpto^ 
order cpin'^tfcm^^^w^ the criterioni In thii situiLtion, 4e pi'edictpr 
and i^oaaMtor lose thair laparate identityj and a nama Uke ^'population 
con^l VftiiaUe^ bacom^ more awkward to ttsa. dils Maniple^ Aere 
is tid si^dificant multipla R until die product tan^ || infroduced; ften 
it Jump to values Iflca ,48. This ex^ple is based^ipA sonie of Fiedler's 
^eoenfly reported work on the influence of leadeV^ayman relatiq^ on 
smUl group rfFectiveness/Many of yw may hava see^^&ls writ^n up 
in Time ma^azina ra^ndy, even if you misiad Fiedlers APA prasenta^ 
tion and haven't yet seen his report^to ttia Office of Naval Rdsaarch ( 1 ) * 

Tha gist of it is this. Suppose you ara tiie formally designated lead^ * 
<rf a smtH group or team* Th^e is probaWj^fomeone m the group who 
is your right-hand man— your principal su^ordmate or keyman. If you 
are ttia kind of person who generally has wahri eeUngs for most peo- 
ple^ for whom similaritief aniQpg otfier people are more important ihan 
OTflFeFencaSj it will pay yod=and your group for you to maintain a rela- 
tive, aloofness from your grdupj -and especially from your keyman. On 
the other hand, if yoif tend to tfiipk of otiiers you like as being different 
f rem ^p^a you dislike, it will pi^ Jo do the opposite, namely for you to * 
cultivate sfrong sociomfetric tieswith ymir groups and especially with 
your keyxnam ^ " f- « ; * 

Here^ then^ are two variables you ^^ed to measuiS-to predict a 
particular leader i effecdveness in a ^VtiCular group Wliile neitiier 
Variable is related to the Qritenon, their product hm a^bstantialpega- 
tive coirelation witfi it. Tlfte ^results ih^ppen to be psy^ologically veiy 
Sensible, and they mn pro^lbly be used to counsel leaders towards a 
more effective style of leadership, and ^^redict what groups will re- 
spond best to given styles of leadership. 

TTiese ^o examples have been very different in many respects, but 
they do have two Aings in common in addition to thair featuring of 
important moderator efftflts^^For one tiling they both feature non- 
cognitive variables as predictors— and as moderator^^|f still care 
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which ii which, It ieems to me that we can expect ^^^^ J 
Ais kind relatively inore easily, but cognitive examples are s^U a rea 
p^sibUity. For inStwce. if we could ^^ta^Y ^^"W 

reUabm^ for an individualV test score, in4ependentfy of thejroup 
li A in whih he h appeti^d-WtotedHfr^ould^robabl)^ 



any predictions made from the test score itself., , ^ ,h,diedbv 

lithe second place, both Aese examples were '-t»«»y ^^f • 
breaking a total sample up into subgroups. There f « "'"^y^^^^ 
that ha4 been ^led this far. and t^ ^pmrted 
lack of an organising concept such as themoderator ^ana^M " 
For the past Aree years I have made It a^nt ^ APA^to^o 
seek ouUtudies of this kind; I have always foimd flU.:^^ 
very far aBeld. For some reason, studies InvolvmgJhe „ A"*^^^^*^ - 
; S^nality Syndrome" seem to He in particular heed of som«h.n| like 
WWoderoSr variable. But ho more so. I would say. ^^^^"^ 
co^ural scoring or make clinical Judgments on &e basil of person- 

^ ■ / ^is one last topic on my ag^da. A^suminr you have^ecldg 
■ olSc for moderator effects, wto is the^best way to go abou^^t? 
are at least four methods to consider. 1. You can s art with^ 
Jo^othesis. This is^lways a good idea, 2. If you d.n t haj^ one 
Sndy.^ can look'for one by studying a ^-^V^^^"^^' ^ 
seeking out variables whose interpretatton seems to depend on otl^ 
vl^bL 3. You can do the same thing with larger groups of cases by 
looking to sets of sub-groups within whld» con-^tions sipnfl- . 
mnUy different. 4. You can do the' same thing With items instead of 
vSles, ^y testing the interaction variance of a of ^^^^^l : 
a criterion. This fourth approach is a whole technique m itse f. and I 
wSShere were time to tell you about it. It ccipitai.es on theelc^omc 
^mputers. it can br generalized Ijeyond pairs of items; ""f ' 

old icka of keying patterns of ^sponse to several items into a new 
perspective. / 
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. \1 Metkdd of Factormg Without ^ 
* i ^ ^ CommunaHtiis 



*^ ^ ♦ L / l \ t H U R B T 0 N E 



Eym siBtIt multipla factor analysis was davaloped over twenty years 
a^j^e imkno\m diagon cells of die correliftion matrix have bean a 
s^ois^^^^^blain^ TTif^ oolftQfiimalities and they repre- 

sei^Jtha^art ol tfie toW variance of a test which it shares wlth^one 



lot mbrm of the other tejts in tiie battery* For each method of factoring 
it haS b^tf necessary lo estimate Aeae diagonal values irftiie con-ela^ 
ti^ TOaWx tod various methods have been devised ^for doing so» 

In a theoreti^} case one can construct a correlation matrix of order 
H ani^rank f wh6re f?< The diagonal entries of sudi a correlation^ 
matrix ard €Nn so determined Aat all minors of order (r + 1) van^|^, 
\?V^en Ae or dej n is larger than^ say, 6 or 8, the diagonal values are . * 
usually uniquef ln. dealing witii experimentally observed confelat|on 
~ ^ ^i40oientS| Ae-ooefflcients n have variable anors so Aat a 

;/ low mnk camiot be found in apy exact manner. However, one aan 
usuaUy write another correlation mabrix. widi side eiyxies ^at are 
>^nearly fte same as Ae ex^rimentally observed values and which is of 
a rank mfcch lower ftan n so tiiat, for example f < n/2. This is the 
^ situation often found in tiiultipje factor studies. ^ ? 

Various methods of factoring the correlation matrix have been de- 
vised in which tiie diagonal elements are first estimated. One of the 
siinpleffl^fe assign to each diagonal cell atValue equal to the absolute 
/ value of^ie hi^est entry in the coffesponding column or row. The. 
\ estmiate is revised afier each factor has been exb-acte^-. This rough 
f "method of estipiatfon is quite successful when tiie order i^HfiEtWy^aTge, 
say^ about 15 o|r 20 or higher* Although this procedure is quite success- 
j^'^fnlV foE many seientiflc problems with correlation m^tiices of high 
order^ It is fajr from satlsfactoTy for theoretical formulations of tiie fac- 
toring probl^. yf' - 

TTiereis an important relation between die communalities and *the 
number of factow that are used for describing a coirelation mabrix. 
13iB largitf die numbef of factors, die higher are die communalities. 
• The eommunality of any test variable / is the sum of squares in the " * 
coiresponding row ot the lj||nor matrix If it is decided that the first 
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f ^cipl wees we inad^uate fw tii© description of the OTrrelation 
Am ft© next priridpal aris may be deteraiined. This giv^ 
(r + 1) eoluzana of 4e factor matrix. The oommmiaUty of any test j 
is aen augment^ by dia square of fte enfry in cplumn (r + 1) in the 
If^^pBgnr te flever toasfwii bef oreh and^ow-msny^prineij^l^es^ie^ 
to be detomuied m order to reduce tiie residual confelation coefficients 
to vdii^ tibat be iffftored in terms of their sampUng ereors, Hence 
we have Ae anomdous situation of not knowing at tiie start of Ae 
computations how many factors need to be postulated to account for 
die cowelations. The communaUty estimate should rise wift fte num- 
ber of factors Aat are extracted but 4e relation depends entirely on 
the untoown configuration of test voters in each given problem.J^ 
In practi^ the problem is resolved by adjusting tiie communali^ 
for ea^ factor fliat is extracted from Ae correlations or by repeating 
the whole factoring process until tiie estimated communalities at the 
stMt agree witii flie sum of squares of tiie factor loadings in the rov^s 
of tiie resulting factor mafrix. But this process depends on a certain 
number of factors as determined by the first cycle. The adjustaent 
should be done over again if the investigator decides to increase the 
number of factors used. Wien tiie number of factors is quite large, 
such as 10 or 15, tiien die adjustaients in Ae diagonals are ordinarily 
quite small for each additional factor. 

In toying to relate the theory of multiple factor analysis to practical 
.scientific work with large correlation matrices, Ais situation is evi-^ 
dently quite unsatisfactory, even though the practical compromises 
have been adequate to resolve most of tfie soientiflc problems so far in 
the isolation of die components of human intelligence which have been 
called primary mental abilities. 

Several months ago I was sitting in an airplane in Helsinki in Fin- 
land, waiting for die take-off for Stockholrp. It occurred to me suddenly 
diat diis awkward situation that we have fought and compromised wi& 
for a long tima could be resolved in a ridiculously simple way, I shall 
describe ^the idea here although we have not yet developed the best 
computing methods, and I hope that some other students of the multi- 
pie factor problem will consider the new method and how it might be 
reduced to Ae simplest possible pomputing procedures. 

In the ^^rimentally given correlation matrix, die diagonal cells 
are uritaiown. They are certainly not experimentally given, Let us de- 
termine die first column of die factor matrix by the mediod of least 
squares so Aat die first fartor residuals are a minimum. This is pre- 
cisely what we do in determining the first principal component by 
Hotelling*s iterative method with one important excGption. This except 
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ticm Is Aat we &b i^mown dbigonal mils eomplately, S:bce 

^^^^^•^fly file ^fo^ra^y tmoMna ix»mIation ^ffloien^ pMd^ 
pat© m til© pbsarvatioQ equationj for A© l©ast squarai lolution. \^en 
the fliTtf fTihimii^Qf 

mfi^imlz a r^duab lor Ae'^ilda ^n©latios|, then we find A© fliit 
fitj^or v^du^ A© iide entriei. diagonal cdb ar© ^lor^ 
iMnpleMy, TbM w© proved as before for eadi additional eolumn 
ftdtor maUie un^ the residuals ^ th© sid© conelations ran 
OS ipi^©d, The communalities for Ais glvm niunber of factoii ar© 
then simply tti© sums sguar&s of the rows of Ae factor matite F* 
to tfili jff QonimunaliUes after fte faetoring 

job has b^ oomplet©d. If, for my reason^ we decide to reduce A© 
residuaJj still fm^^, tfiat em be done by exitacting anoth^ factor. 
The new commundities are tiien merely the sums of ^e iquarts of tiie 
rows erf F ^th anoAer ^lumn added. None of Ae factoring need be 
r©p©at^ becaus© ^mmunaUty estmates are ndt involv^ hi Ae 
factoring of fte eorr©lation matrix, ' 

It was seveial w©eks until I had tiie opportunity to by Ae new 
method m FrMil^r^i Geraianyj wlire I discussed this meAod wift 
ftre© of my former students in ^cago* Ttey were Dr, Hans Anger, 
. _ Dr* Stra^.Hmryssoi^ and Rotf Barpnanni Bargmann set up Aree test 
^es md reported ft at the solution gets'^lpse to the principal axes 
solution and Dr. Stan Henrysson has reported amilar flndings with the 
method in Uppsala* When I return to my laboratoiy in Chapel Hill, 
I expect to tavestigate 4e metfiod furftOT, 

The computational prdcedura can be tried to a manner analogous 
to Hotelling^ fterative solution but it is likely tiiat one of several ofter 
alternatives will be moraaffectivep In one mamier of writing fte prob- 
lem we get ftlrd de^ea normal equations which can be solved by 
successive app^idmations with additive coireclions to the assumed 
factor loadings. 

There may be an intarasting geomeWc twist to ftis problOTi to that' 
the customary geometric model may have to be revised but I am not 
prepared yet to elaborate on ttie geometric implications of ftis prob- 
lem* Thm sU^t disturbance of the geometrical model for ttie coCTela- 
tion coefiBcient and tiie factor matrix can be seett wlA a tiieoretical 
case of exact mnKr, When this new me&od is used, thcT tii factor 
residue do not ^^E^h identimlly. They are small, however, "nie 
: : f aQtojmg maUg obiau^d can still be given the usual geomeMcal 
-tate^retation an^die chelation coefflcients are closely represented 
as scalar products of the fift vectorf that are defined by the factor 
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matrix. It is a question to bd Investigated whetiier Ae distitfbancB of 
Ae ttue geomeWcal model is less important ftan fte disturbance jm- 
pUed in aU tte adjusttnents of tiie communalities in practical com- 

puting procedures, ' » a„j„- 

^=i^Thte=woeMn»^»fl9-to-mlnd^^ 
sin^e factor; However. Ae application of such a formula may lead to 
trouble if applied to the residuals for subsequent factors after the HrJ 
factor. Tie flieory of Ais problfm sbould also be investigated with 
reference to tiie possible appearance of Ae Heywood case So far we 
have not encountered it In several trials we find faotorial sohitions 
aiat are close to the principal axes. , . . ^ £a 

T We hope tfiat other students of factorial analysis may be, interested 
to explore this factoring method. 
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PARTICIPANTS 

T. W, ANDERgONs Edward E, CuretoNj Frederic LorDj Frank Rosenblatt, 
; ^ -PvJrRuLd!^ DAVittB* SAuNrifiRs, Robert L. Thorndike, L, 
THURSTONEj Ledyard R Tucker, John TuKEY, 
^ Joseph ZuBiN * , 

Dr. Zubin: Dr. Anderson, did you make each probability of response 
independent or would you allow for interaction, for instance, between 
sayteg yes to OM item md no to anotiberP Is the probability of tiiat 
. jittero based on die product of die original probabilities, or might 
.*^fli^ ^ lom^ jinti^ , 
\ ^^'V^f^^fi^^dNpJim assumption here is that given the latent s^cture 
. spores ,M(rfit tljp ^esp^nses are independent. If one estimates tiie latent 
'^jsa^ametfcrsrM sides of the equations 

I^^V^ndsiytiitJ^ dp nqt correspond so well to the Irit-harid sides, that 
^cfmdil^p^ta;^ assumption is not born# out very well* "nien 
one can eirtaha*^e^mQ4f 1 w instead of basing Ais on 

one latent score, ' you^iiaVfi^^syy^^^ scores. For a particular 

couplet, let us say, you may havfe to throw in an extra latent score to 
account for tfie kind of interaction you are talking about* 

Dr, Ruloni Are there any other questions? I do not know whether 
Dr. Anderson made himself so clear that there are no questions, or so 
obscure that there afe no questions. 

Dr. Curcton; I have a comment that might have a bearing on Dr. 
Zubin's question. Ityis to be nqted that the number of latent classes Js 
so detennined diat tiie responses will be uidependent witfiin sufii, 
classes as far as that can be determined statistically* In other words, 
you smply postulate enough latent classes so that you do have inde- 
pendence witoin each on&of them. - 

Dr. Zubini Is Aere any way of telling how many subgroups you 
have in the overall sampleP ^ 

Dr. ANDmdNr Onie technique for doing that is to cairy over^hat 
you do in factor analysis, again using die analogy between these equa- 
tions. In factor analysis, tiiere are techniques to determine how many 
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factors 4ere we. and ' 

mine how many ^^^9^^^^ ^°^ * »^ wd 

view of fte tiiLetical ifeti^itea^fef t^hP fl«^^ «l 
^ri^d ou^ or Aey do,pdtyply:ai,^ n. ord aary factor 

CtmrroNt Dr. SB^M,!^-itii^:^^y^ P"* 
-moderator variable » tt^ fbrto^jptgltet.: Wdu^^^^ it be equa ly easy 

toput to L square of i^j^U^my^W^^^ ''^'''^ 
stretched firom linear toJpMAbbUcf/f, ;^ , -^^^ Up 

done. Again, in m^^^^V^^:^^^^^^ mcreasmg num= 

mental data. In the absence ftti,^^^^^ t° ^ 

Sese square terms m'ms6m%'m'^^°^}' 
proach to steer clear of lis- ^ op^™ 

exploring these moderator iVWabler JuSt to coinpu e all the 

S^^and nai^ei^^ Wfaad in your basic pred,c- 

tion ente^rise? ■ > A ' ^ . i' ^ * i, ' 

Db SAUNDERS:''»iliis m an agfiroaqh, and it is a,lot of work. , 

DR'ctmBroNr^Yoti^'^WtalW^^ , 

compute an ordin'ar,) ;linVaj/m^^^ for this ,matr« -and take the 

. -coefflcient of det^rtpi^^tiph, (prr^K 
determlnatibn'^ th thftt^f' a; correlation whidr_Ancludes the pr^upt 

. torfelatlbnk^aiilf ,«;res^^ to the usual ndjipn of 

ac«6n ^comlhg^M^'.A^^^^^^^ bf varlj>nceP It s«lns to mefeal^he 

, Bue tfthe ihodef^^ closely 4o:to»teraeti6n 

; spells out iii aispfbach that ytfu can take in testihg the «ff f f « 
• If ^^uct-tel'M n#ed td^mpare these coefBcient|^£ det,rtnn,a.. 

tion wilb 4nd wifiiiiut tiie^p^^^ terms. , . ' ' ■ ■ „ 

^/■T^0mmk:m^ the same thing aS f «;g f ;H°t«lv. 
linVs probeduSi eiccept:^*^^ sum of the squares of ^^}^: 
^^m^^rmoi^^^ of the squares m aU Pj^Je factor. 
S^?^dthl;co^«poridlngr^^^^^^ 

■ pnal "te this we-defti;only with the known values But if you wntg.- 
!^^m^^Stim^,:am^^^^ b« interpreted in an vdinary : 

■ °f °* *^ factor loadings,. 



ERIC 



TESTING PROBLEMS 



133 



E^miANj MOd^, D@pa^i^t of fm- 
* iODQd, New York Ci^ 

, AGO - 

¥mMWF,-My^ynt of Edueatfon, 

Muyland 

Pmmr^ WlUiam D., U^vori^ of North 

V^mso^^ Doaald I&s^nin^ 

Agency Managmeiit AModatioq - 
PBLANp Robot F.p FfudoQtU Isw- 

anoe CompiSky * 
VmMONf C^oxUm A,, Qumm Collage 
FiKiu, Charlai F., Eduoational Teft- 

ing f ervice 
Tmiwm, Barbara^ Educational Testtng 

Service , ' 
Pt.tTvtT EB^ B.» EdumitioDal Te^g 

Pdtjirj A, Tmence; U, S* Afr Fora* 

Mit^d Ara ^ 

PoLLAci* Norman New York Siata 

D^artm^t of CItU Service ' 
P^§§Bir, Sidnay L<, Ohio Stata Univar- 

Pressey, Mfi., Sidoay L., Ohio State 

RABmownZy WiUiam^ B^ik Sbaat 

leg© of EducatioQ^ New YoA City 
Rapasch, Johni Dapartment of Edtica« 

tion^ CoQ^rd^ New Hamp^ire 
Raffabi^, John H.,.Owan^=-Iilinois 
RABEOf j Evel>^, Brooklyn CoUega 
Rasbk, Judith, Quaani Collage 
Raymond^ l^omai J*, HarvarfjUnlver- 

RAi^OHDj Vincent R.g Newark Collage < 

of Engineering 
R£AD, "^omai,^ Paddle Sc^l, pghts- 
_ towi. New Jariey 
RsGANf Jamas J^^ Specid Davicai Cen^ 

tar, ONR 
Ri3^n^» MalviUp Qua^s Coyege 
RsMKffiBSj H. H*, Purdue University 
RmmR, W, H., Educational Testing 

Service _ 
RHui^i Wanen A., Armad Forc^ 



RiCTAB^OH, W,p Ridii^dson, Bel- 
lo^i ttamy-^d Company, In^rpO'^ 
rated ^ 1 ^ 

RiCTARKOK, Ru^ P/ NYSPA, Naw 
York City 

RiOES, J, H„ Jr,, Piychotogi^ Col- 
lation 

RmkLo^^ Jade K*, EducmUonai 

ing Sendee 
RoBBms, ^ing. Queans CoUaga 
RooA^ Fablo, Deparbn^t of Educatiloa, 

Fuarto Hira 
Roq^ Robert T., Jr., Fordham Univer- 

RoiENBLATT, FraiJc, GornaU UniVOTity 
RosNBR^ Benjamin, Columbia Univetiity 
RuLON, ^P. J.i Haowrd UnivOTi^ 
Sadacca, Robert, Educational Tasting 

Service > 
%AwmmBi David, R,, Education^ Test^ 

ing Sa^ca 
Satt, Ewaid, Rensielaer PolytedbniG 

Institute ' 

Sawin, E, Air FoTO ROTC HSd-, 
quartan, Montgom^ 

ScATO, Alio^ U. a. Office of Edu- 
cation. 

BchTm^ Douglaj E,, American Sodal 

Hygiana As^idation 
ScHAPffiO, Harold B., Great Nedc, New 

York 

ScHRAnER, William B., Educational 
Testing Service 

Scott, C. Winields Vocational Counsel* 
ing Saivices ln6* 

Seashoae, Harold G., Psychological 
Coiporation 

Sebald» Dorothy D., Hunter CoQege^ 

Sebau^ Ji F., Worthington Corporation' 

Seibm^p Dean W., Harvard University 

Sfohza, Ridiard F.,, New York State 
Dapartment of Civil Service ■ ^ ' ^ 

Sharp, Catherine C, Edocational Test- 
ing Servic© 

Shaycoft, Madon F*, American Insti- 
tute for Researeh 

Smio^d, Benjamin, Educational Tak- 
ing Seryiee . ^ 

SsGMANK, P, T., Ameri^n Tdephone 
& Tele^ph 



Us 



1% 



134 1954 iNVITATIONAL CONFERENCE 

So^miAK, Ha^, City Qoi%gm of SwmEi?oro, Francei, Educational Test- 

New YoA ^ ^ ing S^cm 

SaiPiOK/ Mil. El^ferth a;, fflinofa la- SYMONBi, Perdval M., Teachefi CoU 

ititute of T^tobgy ^ legl, (^lumbia University 

B%ms^ iUlan B., Uidymity of Con- Tasso* Charl^ A*, American Cyanamid 

ne^cut ^ ; Tatiuoea, Maurice,- Harvard Univer- 

^ym^¥, Maijorie B,, K™t®f CoUega - dj^ 

Smit, Jo Anne, Waitington, C. Taylob, John F,, Woodmere Academy 

Smtth, denzd D., Office of Naval Re= Taylor, Justine N., Educational Teit 

searob ! / Ing Service 

SMTm^ Louise, Edu^ationd Testing Templeton, Hugh, New York State 

Service ' ■ ' ■ Departaient of Education , 

SMrrH, Muriel, Edueatiorial Testing Terral, J. E., Educational Testihg 

Service , Service 

Solomon, Robert, Educational Testing THORNDm, Robert L., Teachers CoU 

Service ' ^leg^i Columbia Univenity 

SoRENsoK, Garth, University of Call- Thurston^, L. L,, University of North 

fomia - Carolina p 

SourmB, Man^ T.,' Tower itiU School, TRAjflLER, Arthur E„ Educations Rec- 

Wilmington ordi Bureau 

Sfanby, Emma, Queens College Triggs, Frances, Commission on Diag- 

Spaulding, Geraldlne, Educational Ree- nostic Reading Tests 

drds Bureau Tucker, Anthony C, Department of 

Sfeer, George S., Illinois Institute of Defense « 

Tedmology ' • Tuciro, Ledyard R, Educational Test- 

Spencer, Douglai, New York City ing Service 

Starch, Daniel, Daniel Starch & Staff Tukby, John W., Prince^n University 

' Starch, Mrs» Daniel, Mamaronerk* Tukey, Mrs. John W., Princeton, New 

New York i Je^^^K 

St^:phan, Frederick F., Princeton Vioi- Turnbull, William W., Educational 

vsrsity Testing Service 

Carl, Queens College Twyford, Loran C, Special Devils 

Stovart, Mary, New York University Center, ONB 

Stewart, Naomi, Educational Testing Vickery, Vema L., Southeastern Lou- 
Service isiana College r ^ 
Stice, Glen, Educational Testing Serv^ Voss, Harold A,, Special Devices Cen- 

ice tfi^^ 

Std^, TThomas M., Metropolitan Life Wadell^ Blanddna World Book 

Insurance Company Company * 
Stone, Paul T.," Hunringddn College ' Wahlgren, Hardy L., State Teachers 

Stuart, William A., Educational Test- "College, Geneseo, New York 

ing Seivice WaleeRs^ Helen M., Teachers CioUege, 

Stuldaum, Harold, Metropolitan Life Columbia University 

Insurance Co, Wallace, Wimbum, L., Psychological 

SuLUVAN, Daniel S„ New York State Corporation 

Departeient of Civil Service Walsh, B. Thomas, Personnel Depart- 
SwiiVAN, Richard H,, Educational - ment, Philadelphia 

T^nf Sendee WALSHi John J., Boston CkjUege 

Supm, Donald E„ Teachew College, Walton, Wesley, W/, EdueaUonal 

Columbia UnivfTiity Testing Service 



TESTING 



PROBLEMS _ 135 

— ''- --- -';^r - ■ ■ - 

WttUiu^^ Robert J.J Columbia Univef- 
sity . 

WttLiAMg, Rogar ' K., >Iorgan Stata 

GoUege, Baltimpra 
WttioH, Kenneth M., Princeton Uni«, 

VOTlty ^ 
Wn^Hp FhylUi C«, Queens GoUege 
WiNAHS, S. David, New Jei^ay Depart- 
ment of Education 
Wmoo, Mff^ L.j Virginia State Da- 

pvrbnent of Education * 
WiNTORBOTTOM^ John A^, Educatidnal 

Testing Service 
WiTTBNBOBK, J. R.j Rutgers University 
WOLFL^, paelj American Assoeiatlon 

for the Advancamant of Sdence 
Wc»Dj Ray G,, Ohio State Deparbnenl 

of Eduoatidn 
* WwOHTj Wilbur H., Teachers CoUegaj 

GeneseOp New York 
YoGMANi Harvey, University of Denver 
ZalkikDs' Shaldon S., Brooklyn Collage 
ZoBiHj Josephj Columbia Univenity 



WALTMi^, Hd, New Yqfk State De- 
parbnept of Qvil SerVloe - 
' Watson, Walter S*, ^ Cooper^ Union 
•/Wm^m, A. Unive^^ of New 
'Merico 

WiLNA^ Cecilia T*, Univ^ty of Gen- 

WsKm., Bernica Bamatd Collega 
Wemak/A, P^^qJogical Corpo- 
ration 

Wmtm, Geojg© L„ Silver Burdett 
Company ^ * ' ^ 

WisTLAt Dean K** 'Hansard University 

WmTNBVi^ /^iM Off Ufa InsimnQe 
Agen^ Managamant Assoeiation 

WiiM, MarguKfte M., Board df Edu- 
cadon^ Green^^ ^ 

Wn«, Walt^ a, N*¥ Yorie Univer- 
- - sity — ] * 

\ Wmo, S. S., P^cetofc University 

WtLLAHD, Richard W,, Harvard Univer- 
sity . ^ 



B15R1,S 



ISO 



